Looks ok to me. I suppose the cause of Uncaught exception is being tracked somewhere else ?

I did this 36837ae

for the uncaught exception handler.

Please see if this is useful.

I suppose the cause of Uncaught exception is being tracked somewhere else ?

@rishitesh Please find it here TIBCOSoftware/snappy-spark#2

I did this 36837ae for the uncaught exception handler.

@kneeraj we need to handle the call to SparkUncaughtException inside the Taskrunner and hence Spark layer changes are needed.

Yes. I remembered why I stopped there.

BTW for now it is ok but do you think @hbhanawat a cleaner way will be to delegate the behavior to the Scheduler backend and let them decide what to do with that and the default can be the current behavior in the base class.

I suppose the cause of Uncaught exception is being tracked somewhere else ?

@rishitesh Please find it here TIBCOSoftware/snappy-spark#2

The question is why a CSV formatting error results in an uncaught exception. Isn't that a normal exception that should be sent back to the driver? I think Neeraj mentioned some reason but cannot recall right now. We need to fix that reason first (no exit for uncaught exceptions is also required, of course).

Hemant we should test the same import problem which we saw at citi. All the lines in the csv had some problems and the import failed and it closed the underlying channel between driver and executor and the IOException coming from there was then an uncaught exception which led to system.exit

@kneeraj I tried reproducing the csv issue that you mentioned by specifying incorrect schema but could not. The task and job fails gracefully.

@sumwale as discussed in the meeting, you can have a look at the issue because I don't have context and I am not sure how to repro the channel close issue. This PR handles the uncaught exception issue. We can keep the JIRA open after this PR is fixed to track the channel close issue.

@sumwale @rishitesh @kneeraj please re-review the new code.

Looks ok to me. ( Some conflicts have come )
But we should file a ticket a track the issue of uncaughtexception when spark csv import was failing due to unexpected format in the csv fields and track that.